This invention relates to video encoding and decoding.
One of the recent targets in mobile telecommunications has been to increase the speed of data transmission to enable incorporation of multimedia services to mobile networks. One of the key components of multimedia is digital video. Transmission of video comprises a fairly continuous traffic of data representing moving pictures. As is generally known, the amount of data needed to transfer pictures is high compared with many other types of media, and so far usage of video in low bit-rate terminals has been negligible. However, significant progress has been achieved in the area of low bit-rate video compression. Acceptable video quality can be obtained at bit-rates around 20 kilo bits per second. As a result of this progressive reduction in bit-rate, video will be a viable service to offer over channels such as mobile communications channels.
A video sequence consists of a series of still images or frames. Video compression methods are based on reducing the redundancy and perceptually irrelevant parts of video sequences. The redundancy in video sequences can be categorised into spatial, temporal and spectral redundancy. Spatial redundancy means the correlation between neighbouring pixels within a frame. Temporal redundancy means the correlation between areas of successive frames. Temporal redundancy arises from the likelihood of objects appearing in a previous image also appearing in the current image. Compression can be achieved by generating motion compensation data which describes the motion (i.e. displacement) between similar areas of the current and a previous image. The current image is thus predicted from the previous one. Spectral redundancy means the correlation between the different colour components of the same image.
Video compression methods typically differentiate between images which do or do not utilise temporal redundancy reduction. Compressed images which do not utilise temporal redundancy reduction methods are usually called INTRA or I-frames whereas temporally predicted images are called INTER or P-frames (and also B-frames when the INTER frames may be predicted in a forward or backward manner). In the INTER frame case, the predicted (motion-compensated) image is rarely precise enough and therefore a spatially compressed prediction error image is also a part of each INTER frame.
However, sufficient compression cannot usually be achieved by just reducing the redundancy of the video sequence. Thus, video encoders try to reduce the quality of those parts of the video sequence which are subjectively the least important. In addition, the redundancy of the encoded bitstream is reduced by means of efficient lossless coding of compression parameters and coefficients. The main technique is to use variable length codes.
Compressed video is easily corrupted by transmission errors, mainly for two reasons. Firstly, due to utilisation of temporal predictive differential coding (INTER frames), an error is propagated both spatially and temporally. In practice this means that, once an error occurs, it is easily visible to the human eye for a relatively long time. Especially susceptible are transmissions at low bit-rates where there are only a few INTRA-coded frames (the transmission of INTRA-coded frames would terminate the temporal error propagation). Secondly, the use of variable length codes increases the susceptibility to errors. When a bit error alters a codeword to another one of different length, the decoder loses codeword synchronisation and also decodes subsequent error-free codewords (comprising several bits) incorrectly until the next synchronisation or start code. (A synchronisation code is a bit pattern which cannot be generated from any legal combination of other codewords.)
One of the inherent characteristics of wireless data transmission is a relatively high bit error probability. This problem can be addressed by various transport, network and link layer retransmission schemes. However the drawback of such schemes is the possibility of unlimited and fluctuating transmission delays. In conversational audio-visual services, it is unacceptable to have large end-to-end delays. Thus retransmission schemes cannot be used in such services. Instead one must try to detect and conceal the transmission errors. In streaming audio-visual retrieval services, the transmission delay may vary somewhat due to the fact that some initial buffering occurs before the start of play-back. However, the maximum acceptable transmission delay is fixed and, if exceeded, there is an annoying pause in the play-back. In practice, both reliable and unreliable transport channels are used in retrieval services.
Every bit in a compressed video bitstream does not have an equal importance to the decompressed images. Some bits define vital information such as picture type (e.g. INTRA or INTER), quantiser value and optional coding modes that have been used. ITU-T Recommendation H.263 relates to video coding for low bit-rate communication. In H.263, the most vital information is gathered in the picture header. A transmission error in the picture header typically causes a total misinterpretation of the subsequent bits defining the picture content. Due to utilisation of temporal predictive differential coding (INTER frames), the error is propagated both spatially and temporally. Thus, a normal approach to picture header corruption is to freeze the previous picture on the screen, to send an INTRA picture request to the transmitting terminal and to wait for the requested INTRA frame. This may cause an annoying pause in the received video, especially in real-time conversational video sequences.
Transmission errors have a different nature depending on the underlying network. In packet-switched networks, such as the internet etc., transmission errors are typically packet losses (due to congestion in network elements). In circuit-switched networks, such as mobile networks (e.g. HSCSD for GSM), transmission errors are typically bit errors where a ‘1’ is corrupted to ‘0’ or vice versa.
To impede degradations in images introduced by transmission errors, retransmissions can be used, error detection and/or error correction methods can be applied, and/or effects from the received corrupted data can be concealed. Normally retransmission provides a reasonable way to protect video data streams from errors, but large round-trip delays associated with low bit-rate transmission and moderate or high error rates make it practically impossible to use retransmission, especially with real-time videophone applications. Error detection and correction methods usually require large transmission overheads since they add some redundancy to the data. Consequently, for low bit-rate applications, error concealment can be considered as a preferred way to protect and recover images from transmission errors. Video error concealment methods are typically applicable to transmission errors occurring through packet loss and bit corruption.
H.263 is an ITU-T recommendation of video coding for low bit-rate communication which generally means data rates below 64 kbps. The recommendation specifies the bitstream syntax and the decoding of the bitstream. Currently, there are two versions of H.263. Version 1 consists of the core algorithm and four optional coding modes. H.263 version 2 is an extension of version 1 providing twelve new negotiable coding modes. H.263 is currently one of the most-favoured coding methods proposed for mobile wireless applications, where the bit rate is of the order of 28.8 bits per second and Quarter Common Intermediate Format (QCIF) pictures of 176×144 pixels are usually used. Currently the expected bit rates for third generation wireless products is around 64 kbps and the image resolution may be higher.
Pictures are coded as luminance (Y) and two colour difference (chrominance) components (CB and CR). The chrominance pictures are sampled at half the resolution of the luminance picture along both co-ordinate axes. Picture data is coded on a block-by-block basis, each block representing 8×8 pixels of luminance or chrominance.
Each coded picture, as well as the corresponding coded bitstream, is arranged in a hierarchical structure with four layers, which are from bottom to top: block layer, macroblock layer, picture segment layer and picture layer. The picture segment layer can either be arranged as a group of blocks or a slice.
A block relates to 8×8 pixels of luminance or chrominance. Block layer data consists of uniformly quantised discrete cosine transform coefficients, which are scanned in zigzag order, processed with a run-length encoder and coded with variable length codes.
Each macroblock relates to 16×16 pixels of luminance and the spatially corresponding 8×8 pixels of the two chrominance components. In other words, a macroblock consists of four 8×8 luminance blocks and the two spatially corresponding 8×8 colour difference blocks. Each INTER macroblock is associated with a motion vector which defines the position of a corresponding area in the reference frame which resembles the pixels of the current INTER macroblock. The INTER macroblock data comprises coded prediction error data for the pixels of the macroblock.
Usually, each picture is divided into segments known as groups of blocks (GOBs). A group of blocks (GOB) for a QCIF (Quarter Common Intermediate Format) picture typically comprises one row of macroblocks (i.e. 11 macroblocks). Data for each GOB consists of an optional GOB header followed by data for the macroblocks within the GOB.
If the optional slice structured mode is used, each picture is divided into slices instead of GOBs. A slice contains a number of consecutive macroblocks in scan-order. Data for each slice consists of a slice header followed by data for the macroblocks of the slice.
The picture layer data contain parameters affecting the whole picture area and the decoding of the picture data. The coded parameter data is arranged in a so-called picture header. In QCIF format a picture is divided into 176×144 pixels which corresponds to 9 rows of 11 macroblocks.
Picture and GOB (or slice) headers begin with a synchronisation or start code. No other code word or a legal combination of code words can form the same bit pattern as the synchronisation codes. Thus, the synchronisation codes can be used for bitstream error detection and for resynchronisation after bit errors.
H.263 is the video compression standard used in the ITU-T Recommendation H.324“Terminal for Low Bit-Rate Communication” February 1998, which defines videophone communication over PSTN and mobile networks. When a H.324 connection is run over a wireless channel, it is likely that the received bitstream contains transmission errors. In a H.263 video bitstream, these errors are extremely harmful if they occur in picture headers. Such an error may prevent the decoding of the picture contents. Errors in INTRA picture headers cause the most severe implications, since these pictures are used as initial temporal prediction sources. Errors in an INTRA picture header detrimentally affect the corresponding decoded INTRA picture and each subsequent picture initially predicted from this INTRA picture.